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¢ Characterizing calorie burn during swimming and using learning algorithms 
to tune the functionality to individual differences 


¢ Developed novel experimental hardware and tested on 700 swimmers 


¢ To develop a feature on one app for the Apple Watch 
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Orbiter Damage Inspection 


Spatial Sind Observer 


e Simple engineering tool to measure ee Target I identification 
target visibility 


Replaces human observer in —s = 
systems engineering ; 
Based on science model 5 EE Pe 


Prototype available 
Patented 

e Users include FAA, ARL, industry Image compression 
e Wide range of applications 


Aircraft/UAV Detection 
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. a NASA "See and Avoid" Visibility Calculator 
Night Vision Systems —_ 
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Watson, A. B., & Ahumada, A. J., Jr. (2005). Spatial Standard Observer for 
Visual Technology. Paper presented at the IEEE International Conference 


Algorithm Overview 
9/11/17 on Systems, Man, and Cybernetics 8SMC). 


Path to Collaborative, Human-in-the-Loop Planning Systems 


Mars Rovers 
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boratory Irterface 


Mars Exploration Rover: 
Maestro & MAGPEN 
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Phoenix 
Science Interface 


Phoenix Lander 


Power Planning 
& Analysis Tool 
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Analog Mission 
Self-Scheduling 


Playbook 


Mars 
2020 


2015 


ISS 
Self-Scheduling 


Intelligent Systems and Humans 


Fundamental difference between developing machine learning to 
effectively support human problem solving and interfacing with human 


problem solvers. Work needed on both challenges. 
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Aircraft 


Automation 


Increasingly 
Autonomous 
System ‘“ 


Aircraft State 


* System 


Human Faults 


Autonomy 
Teaming eather 


& Traffic 
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Human/Machine Interface 
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Independent of how a 
system communicates 
with humans, its core 
functionality needs to 
be designed and around 
human capabilities, yet 
we don’t always know 
what those are. 


From SECAT briefing 
package, Aponsonet al., 
09/2016 
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A 
3 What do we need to know about ~ 
; how humans solve problem so sas se 


Three Underlying Assumptions 


1. Machine learning is progressing down a different path 
than human learning 


2. Humans are both the limiting and enabling elements of 
complex functional systems 

3. Need to consider nominal and off-nominal situations 
separately 


Machine Learning Path 


- Machine learning is progressing down a different path 


than human learning 


PROGRESS 


2005 


2010 


2015 


Pattern analysis, 
image recognition 
(e.g., NLP, SDCs, 
Go, Watson) 


General problem 
solving, adaptive 
reasoning, 
knowledge-level 
learning 
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Self-Driving Cars 
Driving is a low cognitive demand activity (most of the time) 


¢ About 5 years of SDC trials 
¢ Currently at around 1 critical disengage per 40,000 miles 


¢ Humans Drivers: 
¢ 1.2 fatal accidents per 100,000,000 miles driven 
¢ 99 injury accidents per 100,000,000 miles driven ~ 1 injury accident per 
1,000,000 m 


¢ Control Center provides high-level goals when vehicle requires assistance 


¢ Time from software hand-back to human control ~ 1min 


¢ Vehicles can drive paths they “know” 
¢ Vehicle responsible for own safety 
¢ Vehicle control not handed back to human in emergency 


Humans as Limiting/Enabling 


- What kinds of problems are amenable to current 
machine learning approaches: 
- NLP and translation 
- Recognition/classification tasks (e.g., melanoma detection) 
- Problems with Scarce data: 
- Challenger, Columbia 
- Many have both: e.g., driving 


Human learning system appears optimized for learning from 
little data combined with strong induction. E.g., language 
acquisition. Pilots report solving unexpected safety issues 20% 
of flights. 
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Knowledge 


Relative strengths of computer vs. human information processing 


From Cummings, M.L., " 


Expertise 


Uncertainty 


Humans 


" IEEE Intelligent Systems, (2014) 29(5), p. 62-69. 


e.g, Current HITL 
for ATM Next-Gen 
research 


Adaptive Expertise 


Knowledge 


Uncertainty 


Skills 


Relative strengths of computer vs. human information processing 


Architecture based on autonomy performing all skill and rule-based roles, as well as most knowledge-based roles. 
Manpower reduced by two orders of magnitude with remaining expert humans teaming with machine intelligence to 
solve complex problem solving under uncertainty. Machine intelligence for airspace management evolves from the 
outset to support teaming with small set of expert humans to support cooperative problem-solving. 


Human Expertise & Problem-Solving 


“In an information-rich world, the wealth of 
information means a dearth of something else: a 
scarcity of whatever it is that information consumes. 
What information consumes is rather obvious: 


— Herbert A. Simon 


Human Expertise & Problem-Solving 


“In an information-rich world, the wealth of 
information means a dearth of something else: a 
scarcity of whatever it is that information consumes. 
What information consumes is rather obvious: it 
consumes the attention of its recipients. Hence a 
wealth of information creates a poverty of attention 
and a need to allocate that attention efficiently 
among the overabundance of information sources 
that might consume it.” 


— Herbert A. Simon 


Ames Research Center nasa 


¢ Application of Gibsons’ 
Ecological Psychology 


¢ Alternatives to using 
human central attention 
resource 


¢ Acar more like a horse 


Teaming of Human and Machine Intelligence 


¢ Even as computers get very “intelligent”, it is very likely that the nature 
of the their intelligence will be different than that of humans (unless they 
become omniscient or we program them to function just like humans) 


¢ Humans are particularly good at adaptive problem-solving and discovery, 
areas where there has been little machine intelligence progress 


¢ Successful efforts going forward will be those that wrap new machine 
intelligence capabilities around human competencies in order to get the 
most out of each 


Goal for Human-Systems Integration Researchers: 


Help characterize those aspects of human performance that 
will allow the enabling capabilities of the human to function 
effectively when teamed with machine intelligence. 
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Final Thoughts 


¢ Humans will remain important components of complex 
systems 


¢ Use human adaptive expertise as much as possible 


¢ Understand where humans are limiting components of 
system performance and focus machine intelligence there 
¢ Non-Bayesian 
¢ Limited working memory 
¢ Low, slow access to long-term memory 
¢ Single-threaded attention 
¢ Systematic reasoning biases 


¢ Be aware of areas where you don’t have big data 


¢ Don’t assume pattern association can solve all problems 
(Not all problems are associative in nature) 
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Thank you 


